*\Do file paper ""The Behaviour of Populist Parties in Parliament. The Policy Agendas of Populist and Other Political Parties in the Italian Question Time" "
*\Alice Cavalieri, Caterina Froio
*\Changes 21 05 2021

use "C:\Users\cater\Dropbox\Paper SESP\Paper_entropy\RISP_II round\Data\dataset02032021.dta" /*replace with yours*/

** Final analyses: run using The PopuList to identify "populist" and "Other" parties (https://popu-list.org/) 

**Generate nr of parliamentary questions per legislature 
generate npqleg=.
replace npqleg=1007 if legislature==13
replace npqleg=1111 if legislature==14
replace npqleg=716 if legislature==15
replace npqleg=1118 if legislature==16
replace npqleg=1722 if legislature==17
replace npqleg=292 if legislature==18

**Generate nr of party groups per legislature
*\bigtab group legislature
*\generate nparl=.

*legislatures where only Lega exists, included in the count
*\replace nparl=16 if legislature==13
*\replace nparl=8 if legislature==14
*\replace nparl=14 if legislature==15
*\replace nparl=9 if legislature==16

*legislatures where Lega and M5S exist, included in the count
*\replace nparl=48 if legislature==17
*\replace nparl=13 if legislature==18


**generating var for party groups with two categories: 'populist' and 'other' parties 
label define populist 0 "Other party" 1 "Populist" 
label values populist populist

**Revisions2: correct categorization of populist and non-populist parties
replace populist=0 if group=="ALLEANZA NAZIONALE"
replace populist=0 if group=="ITALIA DEI VALORI"
replace populist=0 if group=="RIFONDAZIONE COMUNISTA"
replace populist=0 if group=="SINISTRA ECOLOGIA LIBERTA'"

**generating var for party groups with two categories: 'populist' and 'other' parties  when in gvt/opp
gen group3=.
replace group3= 1 if thepopulist==1 & gov==1
replace group3= 2 if thepopulist==1 & gov==0
replace group3= 3 if thepopulist==0 & gov==1
replace group3= 4 if thepopulist==0 & gov==0
label define group3 1 "populist gov" 2 "populist opp" 3 "other gov" 4 "other opp" 
label values group3 group3


**Figure 1: nr of pq by party group // Question to answer: Do populist behave differently from other parties "quantitatively", i.e. in terms of nr of pq submitted?

*Calculate % of pq asked by "other" and "populist" parties per legislature
generate pq=1  
egen totpq= sum(pq), by(legislature) //tot nr of pq by leg
egen totpqlg = sum(pq),by(legislature thepopulist) // nr of pq by leg and populist/other party
gen sharepqlg=totpqlg/totpq*100 //share of pq by pop/other by leg


*calculate average nr of pq asked by "populist" and "other" parties per legislature
gen sharepq_oth=sharepqlg if thepopulist==0
gen sharepq_pop=sharepqlg if thepopulist==1


/*graph Figure 1 : nr of pq by party group
twoway connected sharepq_oth sharepq_pop  legislature

twoway                    ///
(connected sharepq_oth legislature, mcolor(black) msize(medsmall) lcolor (black) lwidth(medthick)) ///
(connected sharepq_pop  legislature, mcolor(black) msize(medsmall) lpattern(shortdash) lcolor (black) lwidth(medthick)), ///
subtitle(" ", margin(medium)) graphregion(ifcolor(white)) ///
ytitle("% of parliamentary questions", margin(medium))  ylabel(10 20 30 40 50 60 70, ///
angle(horizontal)) ///
xtitle("Legislature", margin(none)) xlabel( 13 14 15 16 17 18, angle(45) labsize(small)) ///
legend(label(1 "Populist"))  ///
		 legend(label(2 "Others"))  ///
		 legend(lstyle(none)) ///
		 legend(size(small)) ///
		 legend(symysize(4)) ///
		 legend(symxsize(11))

Results no difference! */ 


**Figure 2: Distribution of attention across issues //// Question to answer: Do Populist behave differently from other parties in terms of issue attention?

*selection of ten most frequent topics 
gen top10=.
replace top10=1 if majortopic== 1
replace top10=2 if majortopic== 3
replace top10=3 if majortopic== 5
replace top10=4 if majortopic== 6
replace top10=5 if majortopic== 7
replace top10=6 if majortopic== 9
replace top10=7 if majortopic== 10
replace top10=8 if majortopic== 12
replace top10=9 if majortopic== 15
replace top10=10 if majortopic== 20

label define top10 1 "Macroeconomics" 2 "Health" ///
3 "Labor & Employment"  4 "Education" 5 "Environment" 6 "Immigration" ///
7 "Transportation" 8 "Law & Crime" 9 "Banking & Finance" ///
10 "Government Operations" 
label values top10 top10


egen X= sum(pq),by(thepopulist top10)
egen Y= sum(pq),by(thepopulist)
gen attshare=X/Y*100
 
*share of attention by populist, Other party
gen att_pop=attshare if thepopulist==1
gen att_oth=attshare if thepopulist==0


*graph Figure 2 : Distribution of attention across issues
graph dot att_pop att_oth, over(top10)

*RESULTS DON'T CHANGE MUCH


**Figure 3: Distribution of attention across issues  / Question to answer: distribution of attention by pop/nonpop govt/opp

*Generating var for populists and non-populists, in govt/opp
gen govpopnonpop=.
replace govpopnonpop= 1 if group3==1 
replace govpopnonpop= 2 if group3==3
label define govpopnonpop 1 "populists in gov" 2 "nonpop in gov " 
label values govpopnonpop govpopnonpop 

gen oppopnonpop=.
replace oppopnonpop= 1 if group3==2
replace oppopnonpop= 2 if group3==4 
label define oppopnonpop 1 "populists in opp" 2 "nonpop in opp" 
label values oppopnonpop oppopnonpop

*generate percentage scores for parties in gov
egen Xgov= sum(pq),by(govpopnonpop top10)
egen Ygov= sum(pq),by(govpopnonpop)
gen attsharegov=Xgov/Ygov*100

gen popgovPerc=attsharegov if govpopnonpop==1
gen nonpopgovPerc=attsharegov if govpopnonpop==2

*graph Figure X : Distribution of attention across issues Pop/nonpop in govt
graph dot popgovPerc nonpopgovPerc, over(top10)
graph save go.gph 

*generate percentage scores for parties in opp
egen Xopp= sum(pq),by(oppopnonpop top10)
egen Yopp= sum(pq),by(oppopnonpop)
gen attshareopp=Xopp/Yopp*100

gen popoppPerc=attshareopp if oppopnonpop==1
gen nonpopoppPerc=attshareopp if oppopnonpop==2

*graph Figure X : Distribution of attention across issues Pop/nonpop in opp
graph dot popoppPerc nonpopoppPerc, over(top10)
graph save op.gph


* Combine graphs paries in gov and in opposition
graph combine go.gph op.gph, ycommon col(2)
graphregion(color(white))



**Run significance tests. use t-tests because chi2 cannot be used with percentages

*independent t test

// is the diff between attention of populists and nonpop significant? (NEW RESULTS YES)
ttest attshare, by(thepopulist) unequal
// is the diff btw attention of populists and nonpop in govt significant? (NO)
ttest attsharegov, by(thepopulist) unequal
// is the diff btw attention of populists and nonpop in opp significant? (YES)
ttest attshareopp, by(thepopulist) unequal

// OPTIONAL is the diff btw attention of govt and opp significant? (NO)
ttest attshare, by(gov) unequal


STOP HERE! now start entropy calculations

**Calculations Shannon Diversity Index (H) ENTROPY** 
*(TO RUN THESE COMMANDS USE DATASET 1 BUT SAVE ON A SEPARATE FILE/DIFFERENT NAME BECAUSE THE COLLAPSE COMMAND SUBSTITUTES THE ORIGINAL DATATSET) 
 
*generate dummies for each majortopic
tabulate majortopic, generate(capdum)

*aggregate dummies by each party in each legislature 
collapse(sum) capdum* , by (legislature thepopulist)
egen totqt = rowtotal(capdum*)

*(FROM HERE USE DATASET CavalieriFroio_(collapsed for Entropy calculation).dta

*turn it into percentages: generate variables with proportion of attention given to each Majortopic by each party in each legislature
foreach var of varlist capdum1-capdum21 {
  generate p`var' = `var'/totqt
}


*calculation of Entropy: generate Shannon Entropy score per party-legislature 
foreach x of varlist pcapdum* {
   replace `x'= 1 if `x'==0 
}
gen Entropy = -(pcapdum1*ln(pcapdum1) + pcapdum2*ln(pcapdum2) + pcapdum3*ln(pcapdum3) + pcapdum4*ln(pcapdum4) + pcapdum5*ln(pcapdum5) + pcapdum6*ln(pcapdum6) + pcapdum7*ln(pcapdum7) + pcapdum8*ln(pcapdum8) + pcapdum9*ln(pcapdum9) + pcapdum10*ln(pcapdum10) + pcapdum11*ln(pcapdum11) + pcapdum12*ln(pcapdum12) + pcapdum13*ln(pcapdum13) + pcapdum14*ln(pcapdum14) + pcapdum15*ln(pcapdum15) + pcapdum16*ln(pcapdum16) + pcapdum17*ln(pcapdum17) + pcapdum18*ln(pcapdum18) + pcapdum19*ln(pcapdum19)+ pcapdum20*ln(pcapdum20) + pcapdum21*ln(pcapdum21)) / ln(21)

//*calculation of Entropy: generate Shannon Entropy score per party-government/opposition
*(TO RUN THESE COMMANDS GO BACK TO DATASET 1) 
*use: C:\Users\cater\Dropbox\Paper SESP\Data\CavalieriFroio.dta
*generate dummies for each majortopic
tabulate majortopic, generate(capdum)


*aggregate dummies by party-legislature 
collapse(sum) capdum* , by (legislature group3)
egen totqt = rowtotal(capdum*)

*turn it into percentages
foreach var of varlist capdum1-capdum21 {
  generate p`var' = `var'/totqt
}

*calculation of Entropy
foreach x of varlist pcapdum* {
   replace `x'= 1 if `x'==0 
}
gen Entropy = -(pcapdum1*ln(pcapdum1) + pcapdum2*ln(pcapdum2) + pcapdum3*ln(pcapdum3) + pcapdum4*ln(pcapdum4) + pcapdum5*ln(pcapdum5) + pcapdum6*ln(pcapdum6) + pcapdum7*ln(pcapdum7) + pcapdum8*ln(pcapdum8) + pcapdum9*ln(pcapdum9) + pcapdum10*ln(pcapdum10) + pcapdum11*ln(pcapdum11) + pcapdum12*ln(pcapdum12) + pcapdum13*ln(pcapdum13) + pcapdum14*ln(pcapdum14) + pcapdum15*ln(pcapdum15) + pcapdum16*ln(pcapdum16) + pcapdum17*ln(pcapdum17) + pcapdum18*ln(pcapdum18) + pcapdum19*ln(pcapdum19)+ pcapdum20*ln(pcapdum20) + pcapdum21*ln(pcapdum21)) / ln(21)


**Figure Shannon Diversity Index (H) for hypotheses not included in the paper
*(TO RUN THESE COMMANDS GO BACK TO DATASET 3 with values of entropy)
use "C:\Users\cater\Dropbox\Paper SESP\Paper_entropy\Data\dataset_entropyvalues.dta"


*status: 1=aggregate; 2=gov 3=opp
graph dot populist nonpopulist, over(status)

**chi-square tests issues (go to DATASET 2)
tabulate frmajp group2, chi2

**Chi-squared tests entropy (go to dataset 3 with values of entropy)
tabulate status entrop, chi2
tabulate status entrlega, chi2
tabulate status entrmov, chi2

*TO DO 
*Figura: Run t test: for the 10 most salient issues by issue (ln-M5s bs other parties
*Figura: Run t-test: su un grafico come Figura 2 distinguendo quando sono al governo e all'opposizione (vedi modello borghetto)/ all issues





STOP HERE, NEXT IS DIFFERENT SPECIFICATION WITH AMENDED the PopuList!

*************************************************************************************************************************************************************
*Analysis run using amending The PopuList to identify "populist" and "Other" parties. We include "Italia dei Valori" and "Alleanza Nazionale"

**Generate nr of parliamentary questions per legislature 
generate npqleg=.
replace npqleg=1007 if legislature==13
replace npqleg=1111 if legislature==14
replace npqleg=716 if legislature==15
replace npqleg=1118 if legislature==16
replace npqleg=1722 if legislature==17
replace npqleg=292 if legislature==18

**Generate nr of party groups per legislature
*\bigtab group legislature
*\generate nparl=.

*legislatures where only Lega exists, included in the count
*\replace nparl=16 if legislature==13
*\replace nparl=8 if legislature==14
*\replace nparl=14 if legislature==15
*\replace nparl=9 if legislature==16

*legislatures where Lega and M5S exist, included in the count
*\replace nparl=48 if legislature==17
*\replace nparl=13 if legislature==18


**generating var for party groups with two categories: 'populist' and 'other' parties 
label define populist 0 "Other party" 1 "Populist" 
label values populist populist


**generating var for party groups with two categories: 'populist' and 'other' parties  when in gvt/opp
gen group3=.
replace group3= 1 if populist==1 & gov==1
replace group3= 2 if populist==1 & gov==0
replace group3= 3 if populist==0 & gov==1
replace group3= 4 if populist==0 & gov==0
label define group3 1 "populist gov" 2 "populist opp" 3 "other gov" 4 "other opp" 
label values group3 group3


**Figure 1: nr of pq by party group // Question to answer: Do populist behave differently from other parties "quantitatively", i.e. in terms of nr of pq submitted?

*Calculate % of pq asked by "other" and "populist" parties per legislature
generate pq=1  
egen totpq= sum(pq), by(legislature) //tot nr of pq by leg
egen totpqlg = sum(pq),by(legislature populist) // nr of pq by leg and populist/other party
gen sharepqlg=totpqlg/totpq*100 //share of pq by pop/other by leg


*calculateaverage nr of pq asked by "populist" and "other" parties per legislature
gen sharepq_oth=sharepqlg if populist==0
gen sharepq_pop=sharepqlg if populist==1


*graph Figure 1 : nr of pq by party group
twoway connected sharepq_oth sharepq_pop  legislature

twoway                    ///
(connected sharepq_oth legislature, mcolor(black) msize(medsmall) lcolor (black) lwidth(medthick)) ///
(connected sharepq_pop  legislature, mcolor(black) msize(medsmall) lpattern(shortdash) lcolor (black) lwidth(medthick)), ///
subtitle(" ", margin(medium)) graphregion(ifcolor(white)) ///
ytitle("% of parliamentary questions", margin(medium))  ylabel(10 20 30 40 50 60 70, ///
angle(horizontal)) ///
xtitle("Legislature", margin(none)) xlabel( 13 14 15 16 17 18, angle(45) labsize(small)) ///
legend(label(1 "Populist"))  ///
		 legend(label(2 "Others"))  ///
		 legend(lstyle(none)) ///
		 legend(size(small)) ///
		 legend(symysize(4)) ///
		 legend(symxsize(11))

* Results no difference! 


**Figure 2: Distribution of attention across issues //// Question to answer: Do Populist behave differently from other parties in terms of issue attention?

*selection of ten most frequent topics 
gen top10=.
replace top10=1 if majortopic== 1
replace top10=2 if majortopic== 3
replace top10=3 if majortopic== 5
replace top10=4 if majortopic== 6
replace top10=5 if majortopic== 7
replace top10=6 if majortopic== 9
replace top10=7 if majortopic== 10
replace top10=8 if majortopic== 12
replace top10=9 if majortopic== 15
replace top10=10 if majortopic== 20

label define top10 1 "Macroeconomics" 2 "Health" ///
3 "Labor & Employment"  4 "Education" 5 "Environment" 6 "Immigration" ///
7 "Transportation" 8 "Law & Crime" 9 "Banking & Finance" ///
10 "Government Operations" 
label values top10 top10


egen X= sum(pq),by(populist top10)
egen Y= sum(pq),by(populist)
gen attshare=X/Y*100
 
*share of attention by populist, Other party
gen att_pop=attshare if populist==1
gen att_oth=attshare if populist==0


*graph Figure 2 : Distribution of attention across issues
graph dot att_pop att_oth, over(top10)

*RESULTS DON'T CHANGE MUCH


**Figure 3: Distribution of attention across issues  / Question to answer: distribution of attention by pop/nonpop govt/opp

*Generating var for populists and non-populists, in govt/opp
gen govpopnonpop=.
replace govpopnonpop= 1 if group3==1 
replace govpopnonpop= 2 if group3==3
label define govpopnonpop 1 "populists in gov" 2 "nonpop in gov " 
label values govpopnonpop govpopnonpop 

gen oppopnonpop=.
replace oppopnonpop= 1 if group3==2
replace oppopnonpop= 2 if group3==4 
label define oppopnonpop 1 "populists in opp" 2 "nonpop in opp" 
label values oppopnonpop oppopnonpop

*generate percentage scores for parties in gov
egen Xgov= sum(pq),by(govpopnonpop top10)
egen Ygov= sum(pq),by(govpopnonpop)
gen attsharegov=Xgov/Ygov*100

gen popgovPerc=attsharegov if govpopnonpop==1
gen nonpopgovPerc=attsharegov if govpopnonpop==2

*graph Figure X : Distribution of attention across issues Pop/nonpop in govt
graph dot popgovPerc nonpopgovPerc, over(top10)
graph save gov.gph 

*generate percentage scores for parties in opp
egen Xopp= sum(pq),by(oppopnonpop top10)
egen Yopp= sum(pq),by(oppopnonpop)
gen attshareopp=Xopp/Yopp*100

gen popoppPerc=attshareopp if oppopnonpop==1
gen nonpopoppPerc=attshareopp if oppopnonpop==2

*graph Figure X : Distribution of attention across issues Pop/nonpop in opp
graph dot popoppPerc nonpopoppPerc, over(top10)
graph save opp.gph


* Combine graphs paries in gov and in opposition
graph combine gov.gph oppo.gph, ycommon col(2)
graphregion(color(white))



**Run significance tests. use t-tests because chi2 cannot be used with percentages

*independent t test

// is the diff between attention of populists and nonpop significant? (NO)
ttest attshare, by(populist) unequal
// is the diff btw attention of populists and nonpop in govt significant? (NO)
ttest attsharegov, by(populist) unequal
// is the diff btw attention of populists and nonpop in opp significant? (NEW RESULTS: NO)
ttest attshareopp, by(populist) unequal

// OPTIONAL is the diff btw attention of govt and opp significant? (NO)
ttest attshare, by(gov) unequal


STOP HERE! now start entropy calculations

**Calculations Shannon Diversity Index (H) ENTROPY** 
*(TO RUN THESE COMMANDS USE DATASET 1 BUT SAVE ON A SEPARATE FILE/DIFFERENT NAME BECAUSE THE COLLAPSE COMMAND SUBSTITUTES THE ORIGINAL DATATSET) 
 
*generate dummies for each majortopic
tabulate majortopic, generate(capdum)

*aggregate dummies by each party in each legislature 
collapse(sum) capdum* , by (legislature populist)
egen totqt = rowtotal(capdum*)

*(FROM HERE USE DATASET CavalieriFroio_(collapsed for Entropy calculation).dta

*turn it into percentages: generate variables with proportion of attention given to each Majortopic by each party in each legislature
foreach var of varlist capdum1-capdum21 {
  generate p`var' = `var'/totqt
}


*calculation of Entropy: generate Shannon Entropy score per party-legislature 
foreach x of varlist pcapdum* {
   replace `x'= 1 if `x'==0 
}
gen Entropy = -(pcapdum1*ln(pcapdum1) + pcapdum2*ln(pcapdum2) + pcapdum3*ln(pcapdum3) + pcapdum4*ln(pcapdum4) + pcapdum5*ln(pcapdum5) + pcapdum6*ln(pcapdum6) + pcapdum7*ln(pcapdum7) + pcapdum8*ln(pcapdum8) + pcapdum9*ln(pcapdum9) + pcapdum10*ln(pcapdum10) + pcapdum11*ln(pcapdum11) + pcapdum12*ln(pcapdum12) + pcapdum13*ln(pcapdum13) + pcapdum14*ln(pcapdum14) + pcapdum15*ln(pcapdum15) + pcapdum16*ln(pcapdum16) + pcapdum17*ln(pcapdum17) + pcapdum18*ln(pcapdum18) + pcapdum19*ln(pcapdum19)+ pcapdum20*ln(pcapdum20) + pcapdum21*ln(pcapdum21)) / ln(21)

//*calculation of Entropy: generate Shannon Entropy score per party-government/opposition
*(TO RUN THESE COMMANDS GO BACK TO DATASET 1) 
*use: C:\Users\cater\Dropbox\Paper SESP\Data\CavalieriFroio.dta
*generate dummies for each majortopic
tabulate majortopic, generate(capdum)


*aggregate dummies by party-legislature 
collapse(sum) capdum* , by (legislature group3)
egen totqt = rowtotal(capdum*)

*turn it into percentages
foreach var of varlist capdum1-capdum21 {
  generate p`var' = `var'/totqt
}

*calculation of Entropy
foreach x of varlist pcapdum* {
   replace `x'= 1 if `x'==0 
}
gen Entropy = -(pcapdum1*ln(pcapdum1) + pcapdum2*ln(pcapdum2) + pcapdum3*ln(pcapdum3) + pcapdum4*ln(pcapdum4) + pcapdum5*ln(pcapdum5) + pcapdum6*ln(pcapdum6) + pcapdum7*ln(pcapdum7) + pcapdum8*ln(pcapdum8) + pcapdum9*ln(pcapdum9) + pcapdum10*ln(pcapdum10) + pcapdum11*ln(pcapdum11) + pcapdum12*ln(pcapdum12) + pcapdum13*ln(pcapdum13) + pcapdum14*ln(pcapdum14) + pcapdum15*ln(pcapdum15) + pcapdum16*ln(pcapdum16) + pcapdum17*ln(pcapdum17) + pcapdum18*ln(pcapdum18) + pcapdum19*ln(pcapdum19)+ pcapdum20*ln(pcapdum20) + pcapdum21*ln(pcapdum21)) / ln(21)


**Figure Shannon Diversity Index (H) for hypotheses not included in the paper
*(TO RUN THESE COMMANDS GO BACK TO DATASET 3 with values of entropy)
use "C:\Users\cater\Dropbox\Paper SESP\Paper_entropy\Data\dataset_entropyvalues.dta"


*status: 1=aggregate; 2=gov 3=opp
graph dot populist nonpopulist, over(status)

**chi-square tests issues (go to DATASET 2)
tabulate frmajp group2, chi2

**Chi-squared tests entropy (go to dataset 3 with values of entropy)
tabulate status entrop, chi2
tabulate status entrlega, chi2
tabulate status entrmov, chi2


***USE DATASET dataset_entropylegislature
*
*graph  : entropy figures for populist parties and other parties across the legislatures
twoway connected nonpopulist populist legislature

twoway                    ///
(connected nonpopulist legislature, mcolor(black) msize(medsmall) lcolor (black) lwidth(medthick)) ///
(connected populist  legislature, mcolor(black) msize(medsmall) lpattern(shortdash) lcolor (black) lwidth(medthick)), ///
subtitle(" ", margin(medium)) graphregion(ifcolor(white)) ///
ytitle("Entropy", margin(medium))  ylabel(0 0.5 1, ///
angle(horizontal)) ///
xtitle("Legislature", margin(none)) xlabel( 13 14 15 16 17 18, angle(45) labsize(small)) ///
legend(label(1 "Populist"))  ///
		 legend(label(2 "Others"))  ///
		 legend(lstyle(none)) ///
		 legend(size(small)) ///
		 legend(symysize(4)) ///
		 legend(symxsize(11))



